Differentiable User Models

Probabilistic user modeling is essential for building machine learning systems in the ubiquitous cases with humans in the loop. However, modern advanced user models, often designed as cognitive behavior simulators, are incompatible with modern machine learning pipelines and computationally prohibitive for most practical applications. We address this problem by introducing widely-applicable differentiable surrogates for bypassing this computational bottleneck; the surrogates enable computationally efficient inference with modern cognitive models. We show experimentally that modeling capabilities comparable to the only available solution, existing likelihood-free inference methods, are achievable with a computational cost suitable for online applications. Finally, we demonstrate how AI-assistants can now use cognitive models for online interaction in a menu-search task, which has so far required hours of computation during interaction.Comment: Accepted for publication in The 39th Conference on Uncertainty in Artificial Intelligence (UAI) 202

Towards a Unifying Model of Rationality in Multiagent Systems

Multiagent systems deployed in the real world need to cooperate with other agents (including humans) nearly as effectively as these agents cooperate with one another. To design such AI, and provide guarantees of its effectiveness, we need to clearly specify what types of agents our AI must be able to cooperate with. In this work we propose a generic model of socially intelligent agents, which are individually rational learners that are also able to cooperate with one another (in the sense that their joint behavior is Pareto efficient). We define rationality in terms of the regret incurred by each agent over its lifetime, and show how we can construct socially intelligent agents for different forms of regret. We then discuss the implications of this model for the development of "robust" MAS that can cooperate with a wide variety of socially intelligent agents.Comment: 5 Pages, To appear in the OptLearnMAS Workshop at AAMAS 202

Best-Response Bayesian Reinforcement Learning with Bayes-adaptive POMDPs for Centaurs

Centaurs are half-human, half-AI decision-makers where the AI's goal is to complement the human. To do so, the AI must be able to recognize the goals and constraints of the human and have the means to help them. We present a novel formulation of the interaction between the human and the AI as a sequential game where the agents are modelled using Bayesian best-response models. We show that in this case the AI's problem of helping bounded-rational humans make better decisions reduces to a Bayes-adaptive POMDP. In our simulated experiments, we consider an instantiation of our framework for humans who are subjectively optimistic about the AI's future behaviour. Our results show that when equipped with a model of the human, the AI can infer the human's bounds and nudge them towards better decisions. We discuss ways in which the machine can learn to improve upon its own limitations as well with the help of the human. We identify a novel trade-off for centaurs in partially observable tasks: for the AI's actions to be acceptable to the human, the machine must make sure their beliefs are sufficiently aligned, but aligning beliefs might be costly. We present a preliminary theoretical analysis of this trade-off and its dependence on task structure

Modeling needs user modeling

Modeling has actively tried to take the human out of the loop, originally for objectivity and recently also for automation. We argue that an unnecessary side effect has been that modeling workflows and machine learning pipelines have become restricted to only well-specified problems. Putting the humans back into the models would enable modeling a broader set of problems, through iterative modeling processes in which AI can offer collaborative assistance. However, this requires advances in how we scope our modeling problems, and in the user models. In this perspective article, we characterize the required user models and the challenges ahead for realizing this vision, which would enable new interactive modeling workflows, and human-centric or human-compatible machine learning pipelines

Model-based Multi-agent Reinforcement Learning for AI Assistants

Interaction of humans and AI systems is becoming ubiquitous. Specifically, recent advances in machine learning have allowed AI agents to interactively learn from humans how to perform their tasks. The main focus of this line of research has been to develop AI systems that eventually learn to automate tasks for humans, where the end goal is to remove the human from the loop, even though humans are involved during training. However, this perspective limits the applications of AI systems to cases where full automation is the desired outcome. In this thesis, we focus on settings where an AI agent and a human must collaborate to perform a task, and the end goal of the AI is not to replace human intelligence, but to augment it. AI-assistance for humans involves at least two agents: an AI agent and a human. System designers have no control over the humans, and must develop learning agents that have the capabilities to assist and augment them. To do so, the AI agent must be able to infer the goals, bounds, constraints, and future behaviour of its human partner. In this thesis, we propose a model-based multi-agent reinforcement learning approach, where the AI agent infers a model of its human partner, and uses this model to behave in a way that is maximally helpful for the human.In order to learn a mathematical model of the human from interaction, the AI agent first must have a model space. Since data scarcity is a key problem in human--AI collaboration, defining a model space that is expressive enough to capture human behaviour, yet constrained enough to allow sample-efficient inference is important. Determining the minimal and realistic set of prior assumptions on human behaviour in order to define such model spaces is an open problem. To address this problem, we bring in prior knowledge from cognitive science and behavioural economics, where various mathematical models of human decision-making have been developed. However, incorporating this prior knowledge in multi-agent reinforcement learning is not trivial. We demonstrate that, using the methods developed in this thesis, sufficient statistics of human behaviour can be drawn from these models, and incorporated into multi-agent reinforcement learning. We demonstrate the effectiveness of our approach of incorporating models of human behaviour into multi-agent reinforcement learning in three types of tasks where: (I) The AI must learn the preferences of the human from their feedback to assist them, (II) The AI must teach the human conceptual knowledge to assist them, (III) The AI must infer the cognitive bounds and biases of the human to improve their decisions. In all tasks, our simulated empirical results show that the AI agent can learn to assist the human and improve the human--AI team's performance. Our user study for the case (I) supports the simulated results. We present a theoretical result for case (III) which determines the limits of AI-assistance when the human user disagrees with the AI

Best-Response Bayesian Reinforcement Learning with Bayes-adaptive POMDPs for Centaurs

Funding Information: This work was supported by: the Academy of Finland (Flagship programme: Finnish Center for Artificial Intelligence; decision 828400), the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (grant agreement No. 758824 -INFLUENCE), the UKRI Turing AI World-Leading Researcher Fellowship EP/W002973/1, ELISE travel grant (GA no 951847), KAUTE Foundation, and the Aalto Science-IT Project. Funding Information: This work was supported by: the Academy of Finland (Flagship programme: Finnish Center for Artificial Intelligence; decision 828400), the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No. 758824 —INFLUENCE), the UKRI Turing AI World-Leading Researcher Fellowship EP/W002973/1, ELISE travel grant (GA no 951847), KAUTE Foundation, and the Aalto Science-IT Project. Publisher Copyright: © 2022 International Foundation for Autonomous Agents and Multiagent Systems (www.ifaamas.org). All rights reservedCentaurs are half-human, half-AI decision-makers where the AI's goal is to complement the human. To do so, the AI must be able to recognize the goals and constraints of the human and have the means to help them. We present a novel formulation of the interaction between the human and the AI as a sequential game where the agents are modelled using Bayesian best-response models. We show that in this case the AI's problem of helping bounded-rational humans make better decisions reduces to a Bayes-adaptive POMDP. In our simulated experiments, we consider an instantiation of our framework for humans who are subjectively optimistic about the AI's future behaviour. Our results show that when equipped with a model of the human, the AI can infer the human's bounds and nudge them towards better decisions. We discuss ways in which the machine can learn to improve upon its own limitations as well with the help of the human. We identify a novel trade-off for centaurs in partially observable tasks: for the AI's actions to be acceptable to the human, the machine must make sure their beliefs are sufficiently aligned, but aligning beliefs might be costly. We present a preliminary theoretical analysis of this trade-off and its dependence on task structure.Peer reviewe

Modeling needs user modeling

Funding Information: This work was supported by the Academy of Finland (Flagship programme: Finnish Center for Artificial Intelligence FCAI and decision 345604) Humane-AI-NET and ELISE Networks of Excellence Centres (EU Horizon: 2020 grant agreements 952026 and 951847), UKRI Turing AI World-Leading Researcher Fellowship (EP/W002973/1), and a KAUTE Foundation personal grant for MÇ. | openaire: EC/H2020/952026/EU//HumanE-AI-Net | openaire: EC/H2020/951847/EU//ELISEModeling has actively tried to take the human out of the loop, originally for objectivity and recently also for automation. We argue that an unnecessary side effect has been that modeling workflows and machine learning pipelines have become restricted to only well-specified problems. Putting the humans back into the models would enable modeling a broader set of problems, through iterative modeling processes in which AI can offer collaborative assistance. However, this requires advances in how we scope our modeling problems, and in the user models. In this perspective article, we characterize the required user models and the challenges ahead for realizing this vision, which would enable new interactive modeling workflows, and human-centric or human-compatible machine learning pipelines.Peer reviewe